Optimizing DNN Compilation for Distributed Training With Joint OP and Tensor Fusion

This article proposes DisCo , an automatic deep learning compilation module for data-parallel distributed training. Unlike most compilers that focus on training or inference a single device, optimizes DNN model over multiple GPU machines. Existing single-device strategies do not work well in training, due mainly to communication inefficiency they incur. generates optimized, joint computation operator and tensor fusion enable highly efficient A GNN-based simulator is built effectively estimate per-iteration time achieved by operator/tensor candidates. backtracking search algorithm driven the simulator, navigating efficiently large strategy space identify good minimize time. We compare with existing DL schemes show it achieves speed-up close ideal, full computation-communication overlap case.